9. Execution Infra and cloud costs
This tutorial is a part of the series - Guided set up.
ELT Data runs the data pipelines in customers' cloud environment. The data never leaves the customers' cloud.
ELT Data charges a flat fee for the pipeline execution and the cloud costs are borne by the customer.
ELT Data runs the pipelines using Azure Container Instances or AWS Fargate to keep the costs, maintainenance and management to a minimum. The data is stored in Azure Blob, AWS S3 or GCS.
Pipeline Execution
ELT Data orchestrates, schedules, manages and provides code for the data pipelines. For every pipeline run ELT Data submits a job to a queue. Customer's cloud reads the data from the job queue and in turn spins up servers in customer's cloud for execution.
These servers are spun up on demand, execute the pipeline and are then killed and any data or storage associated with these servers too is cleaned.
The data is persisted on the cloud (AWS S3, Azure Blob, GCS).
Pipeline Execution Server
ELT Data starts with 4GB RAM and 16 GB storage server. We can increase the server size based on customers' data and performance requirements. Please write to contact@eltdata.io to get the correct size of your execution infra.
Pipeline Execution Capacity
The pipeline execution capacity dependes on the quota and organization limits set by your organization. Please schedule a call with contact@eltdata.io to understand how many pipelines can be run concurrently.
Infra Security Policies
AWS - We can setup pipeline execution in any VPC and subnet of your choice. You can keep your production apps and ELT Data execution isolated from each other.
Azure - Pipeline setup can happen in any Resource Group and the permissions allowed to ELT Data executors can be tightly controlled.
Please write to contact@eltdata.io to setup and understand your infra requirements.